Learning Neural Network with Learning Rate Adaptation

In this chapter the analog VLSI implementation of a Multi Layer Perceptron (MLP) network with on-chip learning capability is presented. A MLP architecture is chosen since it can be applied to successfully solve real-world tasks, see among others [33, 6, 9,4]. Many examples of analog implementations of neural networks with on-chip learning capability have been presented in literature, for example [34] [3] [35] [40] [41] [14] [28] [38] [5] [24]. Learning speeds in many of these implementations are rather slow, ranging from milliseconds with volatile weight storage to seconds with non-volatile storagel. Improvements in learning speed require not only dedicated circuit design, but also a careful design of the algorithms and architecture used. This chapter demonstrates the advantages of a local learning rate adaptation algorithm and corresponding hardware implementation for efficient on-chip learning. Design issues regarding analog implementation of on-chip Back Propagation (BP) have been discussed in detail, e.g., in [2, 17, 7]. The learning rate [19] is a critical parameter in learning performance, which has motivated us to consider techniques for local adaptation of learning parameters, the learning rate in particular [8], thus